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What is Claimed: 

1 1 . In a system using a plurality of fixed imagers, a method to create a 

2 high quality virtual image, in real-time, as seen from a virtual viewpoint within a 

3 scene covered by the plurality of fixed imagers, comprising the steps of; 

4 a. selecting at least two images corresponding to at least two of the 

5 plurality of fixed imagers to be used in creating the high quality virtual image; 

6 b. creating at least two depth maps corresponding to the at least two 

7 images; 

8 c. determining at least two sets of warp parameters corresponding to 

9 the at least two images using the at least two depth maps corresponding to said at 
"10 least two images; 

11 d. warping the at least two images to generate at least two warped 

12 images representing the virtual viewpoint using the at least two sets of warp 

13 parameters corresponding to said at least two images; and 

14 e. merging the at least two warped images to create the high quality 

15 virtual image. 

1 2. The method of claim 1 further comprising the step of selecting the 

2 virtual viewpoint based on data supplied by an operator. 

1 3. The method of claim 1 further comprising the step of selecting the 

2 virtual viewpoint based on tracking at least one feature in the scene. 

1 4. The method of claim 1 wherein the steps of creating the at least two 

2 depth maps comprises the steps of; 

3 a. calculating a plurality of optical flow values between the at least 

4 two images; 
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5 b. calculating a plurality of parallax values corresponding to a 

6 plurality of image coordinates in the at least two images from the plurality of 

7 optical flow values; and 

8 c. calculating the at least two depth maps from the plurality of image 

9 coordinates and the plurality of parallax values. 

1 5. The method of claim 1 wherein creating the at least two depth maps 

2 comprises the steps of; 

3 a. mounting a plurality of depth sensing sensors viewing the scene 

4 coincident with the plurality of fixed imagers; 

5 b. selecting at least two depth sensing sensors corresponding to the 

6 at least two images; 

7 c. measuring a plurality of depth values corresponding to a plurality 

8 of image coordinates in the at least two images with said at least two depth sensing 

9 sensors; and 

10 d - creating the at least two depth maps from the plurality of depth 

LI values. 

1 6. The method of claim 1 wherein creating the at least two depth maps 

2 comprises the steps of; 

3 a. separating the at least two images into a plurality of segments, 

4 pixels of each segment having substantially homogenous values; 

5 b. calculating a depth value corresponding to each segment; 

6 c. optimizing the depth values corresponding to each segment; and 

7 d. creating the at least two depth maps from the plurality of 

8 optimized depth values. 
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1 7. The method of claim 1 wherein the step of selecting the at least two 

2 images is based on a proximity of the virtual viewpoint to respective viewpoints 

3 corresponding to the at least two images. 

1 8. The method of claim 1 wherein the step of selecting the at least two 

2 images selects exactly two images. 

1 9. The method of claim 1 wherein the step of selecting the at least two 

2 images selects exactly three images. 

1 10. The method of claim 9 wherein the exactly three images correspond 

2 to three fixed imagers from among the plurality of fixed imagers arranged in a 

3 triangle. 

1 11. The method of claim 1 further comprising the step of placing the 

2 plurality of fixed imagers in a geometric pattern. 

1 12. A virtual camera system to create a high quality virtual image, in 

2 real-time, as seen from a virtual viewpoint, comprising; 

3 a. a plurality of fixed imagers; 

4 b. image selection means for selecting an image from each of at least 

5 two of the plurality of fixed cameras for use in creating the high quality virtual 

6 image; 

7 c. depth estimation means for creating at least two depth maps 

8 corresponding to the at least two images; 

9 d. calculation means for calculating, based on the at least two depth 

10 maps, at least two sets of warp parameters that define respective warpings of the at 

11 least two images to the virtual viewpoint; 

12 e - an image warper which applies the at least two sets of warp 

13 parameters from the calculation means to the at least two images respectively to 

14 create at least two warped images; and 
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15 f. an image merger to merge the at least two warped images to 

16 generate the high quality virtual image. 

1 13. The system of claim 12 wherein the depth estimation means includes 

2 view-based volumetric mapping means to create depth maps of the images. 

1 14. The system of claim 12 wherein the depth estimation means includes 

2 color segmentation depth calculation means to create depth maps of the images. 

1 15. The system of claim 12 further comprising a plurality of depth 

2 sensing sensors aligned to view the scene coincident with the plurality of fixed 

3 imagers, whereby the at least two depth maps are generated using data provided by 

4 the plurality of depth sensing sensors. 

_1 16. A method which uses a plurality of fixed imagers to create at least 

2 one of a mosaic, a three dimensional model of a scene, and a virtual image, 

3 including the step of placing the plurality of fixed imagers in a hexagonal pattern. 

^ 1 17. The method of claim 16, further including the step of deploying the 

; 2 hexagonal pattern of images on a planar surface. 

: 1 18. The method of claim 16 further including the step of deploying the 

= 2 hexagonal pattern of cameras on at least a portion of a tubular surface. 

1 19. The method of claim 16, wherein an angular separation between a 

2 neighboring pair of the plurality of fixed cameras is less than 40° . 

1 20. The method of claim 19 wherein an angular separation between a 

2 neighboring pair of the plurality of fixed cameras is greater than 30° . 

1 21 . The method of claim 16 wherein the hexagonal pattern is elongated in 

2 a direction to provide an aspect ratio equal to an aspect ratio of an image produce 

3 by one of the plurality of fixed imagers. 

1 22. A apparatus to create a local depth map of a scene from a first image 

2 including a first plurality of pixels and a second image including a second plurality 

3 of pixels, comprising the steps of: 
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4 a. segmenting the first image into a plurality of contiguous 

5 segments, each segment having a further plurality of pixels of the first plurality of 

6 pixels, wherein pixel values of the further plurality of pixels are within a 

7 predetermined pixel value range; 

8 b. determining pixel depths of the further plurality of pixels by 

9 comparing the further plurality of pixels to the second plurality of pixels; 

10 c. determining a segment surface based on the determined pixel 

11 depths of the further plurality of pixels; and 

12 d. updating pixel depths of the further plurality of pixels to be within 

13 a predetermined depth range of the segment surface by comparing the further 

14 plurality of pixels to the second plurality of pixels. 

1 23. The method of claim 22, wherein the given segment surface is a 

2 plane. 

1 24. The method of claim 22, wherein step (b) uses an optical flow 

I- 2 method to compare the given plurality of pixels to the second plurality of pixels. 

1 25. A method to create a local depth map of a scene from a first image 

2 showing the scene from a first viewpoint and a second image showing the scene 

3 from a second viewpoint, the first image including a first plurality of pixels and 

4 the second image including a second plurality of pixels, each pixel having a pixel 

5 value, comprising the steps of: 

6 a. dividing the pixel values into a plurality of pixel value ranges; 

7 b. segmenting the first image into a plurality of segments, each 

8 segment including a subset of pixels from the first plurality of pixels selected to 

9 have pixel values within one of the plurality of pixel value ranges; 

10 c. determining a plurality of depth values for the plurality of 

11 segments; 
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12 d. warping the first image into a warped image showing the scene 

13 from the second viewpoint using plurality of depth values; and 

14 e. calculating a first matching score of the second image and the 

15 warped image; 

16 f. hypothesizing that a given segment of the plurality of segments 

17 has a hypothetical depth value equal to a neighboring depth value of a neighboring 

18 segment; 

19 g. calculating a hypothetical matching score of the second image and 

20 a hypothetical warped image based on the hypothetical depth value; and 

21 h. changing a given depth value of the given segment to the 

22 hypothetical depth value, if the hypothetical matching score is better than the first 

23 matching score. 

= ^ 1 26. The method of claim 25, wherein step (c) includes the steps of; 

-3 2 cl . determining pixel depths of a selected subset of pixels 

- 3 corresponding to a selected segment by comparing the selected subset of pixels to 

4 the second plurality of pixels; 

5 c2. determining a selected segment surface based on the determined 

6 pixel depths of the selected subset of pixels; 

7 c3. updating the determined pixel depths of the selected subset of 

8 pixels to be within a predetermined depth range of the selected segment surface by 

9 comparing the selected subset of pixels to the second plurality of pixels; and 

10 c4. determining depth values for all pixels of the selected segment 

11 based on the determined pixel depths of the selected subsets of pixels of the 

12 selected segment surface. 

1 27. The method of claim 25, wherein step (g) includes the steps of; 
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gl . determining a first portion of the second image which corresponds 

to a first visible portion of the further segment in the warped image; 

g2. determining a second portion of the second image which 

corresponds to a second visible portion of the further segment in the hypothetical 
warped image; 

g3. determining a changed portion of the second image which 
corresponds to a union of the first portion and the second portion; 

g4. calculating a changed portion matching score of the changed 
portion of the second image and the warped image; 

g5 . calculating a changed portion hypothetical matching score of the 
second image and the hypothetical warped image; and 

g6. calculating the hypothetical matching score from the first 

matching score, the changed portion matching score, and the changed portion 
hypothetical matching score. 



